FWIII: Characteristics in Local Populations

General Overview

1. Create a scanner which can collect local population data outdoors ✅ and indoors 🚧

   • Get people to walk across the scanner 😬

   • Collect large amounts of shoe pictures w/ time, date, location data 🥿👠👞

2. Create an algorithm which can identify human-recognizable characteristics from scanner images

3. Generate a local database of common footwear features

4. Use local database to characterize frequency of similar shoes in local population

Tasks

Our Assumption in 2018

Types of Computer Vision Tasks

We can reasonably pose this problem in 3 different ways:

Each method requires a different type of labeling schema, data format, etc.

Some are more tedious to generate than others

Initial Approach (~2019)

• Use VGG16 to classify 256x256 px chunks of images

• Goal is to label the entire chunk with one or more classes

VGG16 Shoe Example approach

• Hard to integrate predictions from each chunk back into the main image reliably

Next Approach (2021-2023)

• Try to use object detection

  • Started with FastAI, but had terrible ongoing support

  • Moved to PyTorch, which is a lower level framework
    (Side note: Muxin is amazing at Python)

• New developments:

  • Structured model so that the underlying network was replacable: can swap VGG16 for Resnet50

  • Implementing better metrics, e.g. Intersection over Union for assessing predictions vs. ground truth

Fundamental Problem

• Neural networks are trained on millions of human-annotated real-world photos

• Even shoe soles are artificial relative to a natural scene

• Networks weren’t trained on the artificial patterns or layouts that are used to generate shoes

Pretrained NNs can still generate useful information that is computer-friendly (e.g. Charless’s method)

To work with artificial patterns and get human-like labels, we have to do something different.

Solution?

1. Systematically generate a large library of synthetic data

   • pre-labeled

   • complex characteristics

   • will require several iterations to get right

   • Use to train preliminary model

2. Run 2D patterns through Charless’s network to generate more realistic 3D images

   • Use to train a second-gen model

Solution?

2. Run 2D patterns through Charless’s network to generate more realistic 3D images

   • Use to train a second-gen model

3. Train on Zappos pictures labeled by humans

   • Update 2nd gen model weights (transfer learning)

4. Train on Scanner Photos

   • Update 3rd gen model weights (messy data)

Solution?

Measure performance/accuracy changes over time on a consistent set of stimuli manually derived from real shoes

Synthetic Pattern Generation

• Use SoleMate style labeling for examiner familiarity

Synthetic Data Generation

Region Layout

Synthetic Data Generation

Patterns

Synthetic Data Generation

Outlines

Synthetic Data Generation